Testing the Reliability of Component-Based Safety Critical Software

Testing remains a fundamentally important way to check that a software program behaves as required, but a weakness of testing is that successful testing only leads to informal quality statements. Even where quantitative methods are employed, it is not clear how the objective statements (e.g. 100% code coverage has been achieved) relate to the statements that are really useful such as “the software is correct,” or “the software is reliable.” This inconclusive nature of testing is at the heart of Dijkstra’s famous comment “Program testing can be used to show the presence of bugs, but never to show their absence!” (ref. 1) This paper argues that Dijkstra’s comment is not as important as it might seem, and that software reliability estimates produced by new component-based statistical software testing (CBSST) models provide a testing framework for software quality that is thoroughly formal, but in a different sense to that envisaged by Dijkstra. A significant benefit of these models is that they offer a new verification method for software built with component re-use, based on “proven-in-use” components.